Repository navigation
Give every launch script the same three subcommands - #2431
Conversation
47bc382 to
e6ba544
Compare
1541eac to
f457523
Compare
e6ba544 to
61453a4
Compare
f457523 to
fcf463b
Compare
80d39e6 to
ad9d29b
Compare
0e8b5c8 to
b7eb2ff
Compare
ad9d29b to
51f369d
Compare
b7eb2ff to
cc9fd72
Compare
8566512 to
cb31bb4
Compare
013f985 to
c27ff29
Compare
cb31bb4 to
5c998a4
Compare
5c998a4 to
f1603b0
Compare
598bdda to
f02d568
Compare
f1603b0 to
517a00e
Compare
f02d568 to
9fbd870
Compare
517a00e to
4746701
Compare
9fbd870 to
f096336
Compare
4746701 to
e31022a
Compare
f096336 to
1ddfc61
Compare
e31022a to
18360e3
Compare
fzyzcjy
left a comment
There was a problem hiding this comment.
The patch leaves Kubernetes inference workers without staged node-local models and breaks Inkling's documented CLI. It also leaves the advertised command migration and its documentation incomplete, and these defects persist at deliver-2.
| ), | ||
| U.rsync_cmd(f"{args.model_dir}/{args.model_name}", f"{args.model_local_dir}/{args.model_name}"), | ||
| ] | ||
| return {"trainer": " && ".join(copies)} |
There was a problem hiding this comment.
[P1] Stage checkpoints on inference workers too
When --cluster-backend kubernetes is used with model_local_dir != model_dir (the default here), this map attaches both rsyncs only to the trainer role. In deliver-2, helm_backend/launcher/values/pool_entry.py::_with_prepare_cmd skips inference-engine pods, while compute_engine_launch_cmd passes the local HF path to SGLang as model_path; disaggregated rollout nodes therefore never receive the model, and colocated inference pods can start before the trainer's rsync completes. Add inference-side staging; the same pattern affects the V4/GLM launchers.
| @U.dataclass_cli | ||
| def train(args: ScriptArgs): | ||
| """Run training. Assumes HF checkpoint / torch_dist are already staged.""" | ||
| """Run training; each trainer pod first copies torch_dist to node-local NVMe when the two differ.""" |
There was a problem hiding this comment.
[P1] Keep Inkling's CLI as a command group
After prepare_cp and full_train are removed, this app has exactly one registered command and no callback. Typer collapses such an app into a root command, so the newly documented python scripts/run_inkling.py train ... treats train as an unexpected argument; deliver-2 retains this shape. Restore the common commands or add a callback so train remains a subcommand.
| @U.dataclass_cli | ||
| def prepare_cp(args: ScriptArgs): | ||
| _prepare_cp(args) | ||
| def _prepare_cmd(args: ScriptArgs) -> dict[str, str]: |
There was a problem hiding this comment.
[P2] Add the common prepare command to both V4 launchers
At deliver-2, neither this launcher nor scripts/amd/run_deepseek_v4.py registers a prepare subcommand; both expose only granular prepare-* steps plus train and full-train. Callers relying on this op's advertised three-command contract still get No such command 'prepare', so add a wrapper composing the existing preparation steps (.claude/rules/launch-and-model-scripts.md:63-70).
|
|
||
| @app.command() | ||
| @U.dataclass_cli | ||
| def train(args: ScriptArgs) -> None: |
There was a problem hiding this comment.
[P2] Update every caller of the renamed launcher roles
This split makes train submit-only and removes prepare-cp, but deliver-2 still documents the old interfaces in docs/models/deepseek/deepseek.md:27-38, docs/models/glm/glm4-5.md:60-73, docs/models/deepseek/deepseek-v3-2.md:96-111, and docs/models/glm/glm5-2.md:39-69. The first two commands no longer perform the documented full pipeline, while the latter two fail with an unknown subcommand; update all affected pages as required by .claude/rules/launch-and-model-scripts.md:135-140.
18360e3 to
3e23b91
Compare
19f5f4d to
b5e8d35
Compare
e435ac1 to
b03c433
Compare
0a73e27 to
57d447e
Compare
b03c433 to
4ef9303
Compare
57d447e to
745d6e9
Compare
7511610 to
6cd3a21
Compare
059cf1c to
7ecd119
Compare
2e6b63f to
4b5e930
Compare
7ecd119 to
2d8372d
Compare
4b5e930 to
891fc28
Compare
2d8372d to
24c2cbf
Compare
891fc28 to
d25a600
Compare
63d0a8a to
857c220
Compare
d25a600 to
ea01986
Compare
857c220 to
f78e6aa
Compare
ea01986 to
d9b1c68
Compare
f78e6aa to
857c220
Compare
d9b1c68 to
ea01986
Compare
857c220 to
d4d081d
Compare
ea01986 to
a433af3
Compare
d4d081d to
3eb3093
Compare
3144d0b to
f0c4324
Compare
563f12b to
3eb3093
Compare
Each script had grown its own entry points, so running one meant reading it first. They settle on prepare, train and full_train, with the body split into private steps those three compose, and the recorded command snapshots follow the new subcommand names.
Part of #1837